Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.
Can Your Data Pipelines Be Dangerous?
Extract. Transform. Read.A newsletter from Pipeline: Your Data Engineering ResourceHi past, present or future data professional! Data engineering can be dangerous; ok—not, like, physically, but by building and maintaining data infrastructure, data engineers are given a surprising amount of access and responsibility. Every commit, table alteration and deletion must be made with care. It took 2 years, but I finally learned a shortcut to make developing SQL staging tables less risky and more efficient. Even seemingly minor mistakes like joining on the wrong key can result in losing days or months of valuable data, which can be equal to hundreds of thousands or millions of dollars in revenue visibility. Outside of code mistakes, not paying attention to logistic factors like vendor contracts and API usage can not only result in downtime, in a worst-case scenario it can lead to an all-out blackout. If the stakes sound ominous, I’d suggest examining the root of your hesitation to work more confidently and efficiently—it may even be the code itself. There is a happy medium between freely building data pipelines and using the appropriate guard rails. As long as you take your time and don’t commit code directly to the main branch then you can do data engineering safely and avoid bursting your pipelines. For those who are anti-virus minded, here are this week’s links as plain text:
P.S. Want to learn how to go from code to automated pipeline? Take advantage of my 100% free email course: Deploy Google Cloud Functions In 5 Days. Thanks for ingesting, -Zach |
Pipeline To DE
Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.
Extract. Transform. Read. A newsletter from Pipeline: Your Data Engineering Resource Hi past, present or future data professional! Somewhere along your professional development journey someone lied to you. They told you to crank out resumes because no one reads cover letters. This couldn’t be further from the truth as 87% of hiring managers read cover letters. Such a high read rate represents a compelling opportunity to sell your data skills and showcase a bit of personality. The problem?...
The Latest From Pipeline: Your Data Engineering Resource Hi past, present or future data professional! I hope you’ll indulge my sharing of an important career milestone; unfortunately, I’m not retiring with a gold Rolex snug on my wrist. Instead, this week marks 3 years in data engineering. I’m excited about this work-iversary because it marks a period of time in which I’ve been working in data engineering longer than my prior non-data job. If you’ve read my work you know I’ve taken an...
Extract. Transform. Read. A newsletter from Pipeline: Your Data Engineering Resource Hi past, present or future data professional! Browsing through files recently, I found 100+ old resumes I used to apply for data jobs in 2021. While data science is sold as a “good career”, the truth is it’s always been tough to break in. Those looking for jobs need to do more than ever to distinguish themselves. For anyone looking for a job you may have been taught to network with recruiters and hiring...